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ABSTRACT 



This study investigated whether words that are glossed 
(defined) in a second language text correspond to the words that readers of 
differing ability find difficult. It examined: (1) whether marginal glosses 
provide assistance to students of first-, second-, and third-semester college 
French in their comprehension of authentic text, as measured by a recall 
protocol; (2) what percentage of the glossed material was recalled by these 
students; and (3) whether the words originally glossed in the passage were 
those that seemed problematic to the students. Subjects were 90 university 
students whose native language was English. Each read either a glossed or 
unglossed version of the same text, wrote what they recalled of the text in 
English, and underlined words in the text that they found difficult. Analysis 
of results indicates that the third-semester students did perform better than 
first- or second- semester students, that siabjects with glossed texts did not 
recall significantly more of the glossed material than did the other group, 
and that the textbook authors did gloss many of the words found most 
difficult by the students. Contains 12 references. (MSE) 
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Which Words Should Be Glossed in L2 reading Materials?: 
A Study of First, Second, and Third Semester 
French Students^ Recall. 

by Erin E. Joyce, The Pennsylvania State University 



Introduction: Why Study Glossing? 

The debate over whether glossing actually 
facilitates the reading comprehension of foreign 
or second language learners appears to be not 
unlike a game of table tennis. Whereas some 
research has suggested glosses to be helpful for 
such readers, other studies have challenged their 
effectiveness. Because glossing is a standard fea- 
ture of most foreign and second language text- 
books and materials, it warrants further investi- 
gation. Another related issue that has received 
little attention in the literature is whether or not 
the words that are glossed in a second language 
text correspond to the words that readers of dif- 
fering levels of ability find difficult. The following 
study attempts to provide evidence of whether or 
not students’ indications of unknown items in a 
passage in French coincide with the items that 
have been glossed by the textbook authors in that 
same passage. The following research questions 
will be addressed: 

1) Do marginal glosses provide assistance to 
students of first, second, and third semester 
French in their comprehension of an authen- 
tic text in French as measured by a recall 
protocol exercise? 

2) What percentage of the glossed material is 
recalled by these students as shown in their 
written recall protocols? 

3) Using the same reading passage, are the 
words that were originally glossed in the pas- 
sage the same words that seem problematic to 
these students? 

Review of the Literature 

As far as the debate over whether or not 
glosses aid reading comprehension in foreign lan- 
guage texts, Davis suggested that the use of mar- 
ginal glosses in an authentic literary text in 
French enhanced the comprehension of the pas- 
sage, as determined by a recall protocol task. Luo 
supported Davis’ findings and also suggested that 
glossing is especially helpful in a text containing 
a large amount of unknown vocabulary. 



Replicating Davis’ study, he also came to the con- 
clusion that only glosses written in the subjects’ 
native language were helpful in reading compre- 
hension. 

i 

These two studies countered the results of 
Johnson’s research which suggested the ESL stu- 
dents’ reading comprehension did not benefit 
from receiving glosses under certain conditions 
while reading a text written in English. One 
important factor can be noted about this study, 
however. The text used was not authentic, but 
rather was created by the experimenter for the 
purpose of the study. The researcher may have 
been able to anticipate which words would be dif- 
ficult or easy for the subjects to understand, and 
therefore, could have tailored the text to match 
their vocabulary threshold. Thus, the glosses may 
not have been necessary in aiding comprehension. 

Jacobs and Dufon also reported no significant 
benefit to reading comprehension for students 
who received glosses in an authentic text in 
Spanish. In his review of this study, Luo noted, 
however, that the text, albeit authentic, was taken 
from a second - semester college level Spanish 
textbook, whereas the subjects who participated 
in the study were beyond the second semester 
level. Therefore, the text was probably too easy 
for them, and the glosses were of no use. Such a 
finding supports the need for the text to be diffi- 
cult enough for the readers to require assistance 
in comprehension such as glossing procedures. 

Concerning the related issue of whether or 
not materials developers are choosing to gloss the 
appropriate words and phrases in foreign lan- 
guage materials, Brutten attempted to determine 
whether or not experienced ESL teachers could 
anticipate which words would prove difficult for 
intermediate ESL students in a reading passage. 
Her results showed 91 percent agreement 
between the two groups with respect to the words 
the teachers chose. However, the students did 
label many more words than the teachers had 
anticipated. The implications of this study sug- 
gest that teachers or materials developers may 
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not be glossing enough words even though the 
ones they do gloss relatively correspond to those 
that the students find difficult. She suggests that 
further research should attempt to determine the 
accuracy of students' selection of difficult words 
through a measure of comprehension testing. It 
may be added that such research should also be 
carried out in the context of foreign language 
learning. The current study aims to do this, as 
well as to shed light on the debate over the gener- 
al effectiveness of marginal glossing in foreign 
language texts. 

The Present Study 

Subjects, Ninety subjects participated in this 
study. At the time, all subjects were matriculated 
at The Pennsylvania State University and were 
enrolled in either a first, second, or third semester 
French class. All of the subjects affirmed that 
English was their native language. The French 1, 
2, and 3 curricula at Penn State included much 
practice with authentic^ reading materials, there- 
by ensuring that the subjects were not reading a 
type of material that would be unfamiliar to 
them. Each subject's informed consent was 
obtained on a signed form. 

Ihstruifient, "To determine whether or not 
marginal glosses in a text in French text would 
aid the reading comprehension of students of 
first, second, and third semester French, a recall 
protocol task was chosen for this study (the 
details of the task will be explained in the 
Methods section of this paper). Bernhardt 
claimed that recall protocols “provide a purer 
measure of comprehension, uncomplicated by lin- 
guistic performance and tester interference” 
(200). However, Bernhardt emphasizes that the 
recall protocol should be written in the subjects’ 
native language (200). Lee supported this deci- 
sion in his research saying that writing a recall in 
the target language may be too taxing for stu- 
dents, therefore, more information* may be com- 
prehended than can be recalled in the target lan- 
guage. As the passage for comprehension, a jour- 
nalistic text was selected from the third semester 
French textbook being used at Penn Stated The 
text, entitled “Les Jeunes en France: Adolescents 
plus tot, adultes plus tard” (Young People in 
France: Adolescents Earlier, Adults Later”) was 
chosen for several reasons. First, it consists of 
485 words in glossed version (including glosses). 
Its length might reduce the chance of the subjects' 
being able to rely on their short term memory to 
complete a recall protocol task as suggested by 
Lee & Riley. A second reason for the selection of 



this passage is that even though it was taken 
from an intermediate/advanced French textbook, 
the large number of cognates and high - frequency 
words in the passage make it easier for beginning 
French readers to understand. As noted in the 
review of the literature, glossing appears to be 
more facilitative when the text is difficult enough 
for the readers to require assistance in under- 
standing. Thus, it was presumed that the text 
chosen for the current study would be difficult 
enough to ensure this condition, especially for the 
first semester students. Finally, this passage was 
chosen because it is representative of passages 
that the subjects usually encounter in their 
French classes in terms of length, topic, and 
rhetorical structure. 

Method, The data were collected during the regu- 
lar class periods of the subjects. The subjects 
were told that they were going to be given a cer- 
tain amount of time to read a passage in French 
as many times as they could, and then they were 
going to be asked to write down everything they 
remembered about the text in English on the back 
of the page. One of two versions of^he text were 
distributed to each of the subjects in all three lev- 
els, after their informed consent was obtained. 
One version contained the original glosses that 
were provided by the textbook authors. The other 
version contained no glosses. The first semester 
students were given approximately fourteen min- 
utes to read the passage as many times as they 
could. The second semester students were given 
twelve minutes, and the third semester, ten min- 
utes. These time limits were chosen on the basis 
of previous studies of this type (Carrell; Davis; 
Holley and King). After having read the text, the 
subjects were then given as much time as was 
necessary to write down everything they could 
remember in English on the back of the page. 

In order to treat the third research question 
concerning the words that the subjects found dif- 
ficult in the text, the subjects were asked to per- 
form a second task with the same materials. 
Upon completion of the recall task, the subjects 
were asked to turn their papers over and to 
underline no more than fifteen words in the pas- 
sage of which they would have liked to have been 
given the definitions while they were reading the 
passage. Those with the glossed texts were to 
take into consideration those words that were 
already glossed. They could select those words as 
well as the unglossed words. The limit of fifteen 
words was imposed so that the students would 
give serious thought to which words they selected, 
a strategy resulting in more valid choices. The 
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number fifteen was also chosen because there are 
fifteen glosses in the original text. 

Analysis, Results, and Discussion 

In order to analyze the data, two independent 
variables are identified: the subjects’ semester 
standing in a French class and whether they 
received the glossed or the unglossed text. For 
the first and second research questions, the 
dependent variable is the subjects’ scores on the 
written recall protocol exercise. In order to ana- 
lyze the results for the first research question, 
whether or not the glosses significantly helped 
the subjects to comprehend more information 
from the text as demonstrated by a recall protocol 
exercise, the number of idea units recalled by 
each subject was tallied’. The mean number of 
idea units recalled was then calculated for each of 
the six treatment groups. Table 1 show the mean 
of each recall score for each of the six groupings. 



Table 1 



Means Table for Recall Score 



Semester 


Condition 


n 


Mean 


first 


unglossed 


11 


8.636 


first 


glossed 


12 


9.500 


second 


unglossed 


18 


10.889 


second 


glossed 


17 


14.471 


third 


unglossed 


18 


15.056 


third 


glossed 


13 


13.077 



ANOVA Table for Recall Score 



DF 


Sum of 


Mean F- Value P- Value 




SquEires 


Square 


Semester Level 2 


338.355 


169; 178 2.933 


.0588* 

Glossing 1 


14.449 


14.449 .251 .6180 


Condition 
Semester Level* 2 


125.329 


62.664 1.086 .3422 


Glossing 

Residual 83 i 


4787.426 


57.680 


* Approaching significance at p < .05 



Because the semester level data approached 
significance, a post-hoc procedure, the Fisher 
Protected Least Significant Difference, was per- 
formed, to find where the difference was located. 
The Fisher test revealed that there was a signifi- 
cant difference in the performance of the first 
semester group compared to the third semester 
group, but no significant difference was found 
between the other semester groups, as seen in 
Table 3. 



Table 3 



Fisher’s PLSD for Recall Score 





Mean Diff. 


Grit. Diff. 


P - Value 


First, Second 


-3.542 


4.055 


.0860 


First, Third 


-5.139 


4.157 


.0160* 


Second, Third 


-1.597 


3.726 


.3693 



Effect: semester Level 



Effect: Semester Level * Glossing Condition 



To test for statistical significance, an analysis 
of variance procedure was carried out using a 2x3 
unbalanced factorial design. The results in Table 
2 show that the p-value for the independent vari- 
able of semester level approaches significance. 
However, the difference in glossing condition and 
for semester level vs. glossing condition were not 
significant. 



er|c 



Significance Lrevel: 5% 

* significant 



These statistics would suggest that the third 
semester students did score significantly higher 
than the first semester students, although the 
second semester students did not outperform the 
first semester students, nor did the third semes- 
ter students outperform the second semester stu- 
dents. The analysis also show that the glosses did 
not provide significant help for those students 
with glossed texts. 
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The second research question deals with the 
percentage of the glossed material the subjects 
recalled in their protocols. The recalls were ana- 
lyzed once again and any occurrence of a glossed 
word appearing in the recalls was noted, as 
shown in Table 4. 



Table 4 

Percentage of Glossed Material Recalled 



first 

semester 



second 

semester 



third 

semester 



unglossed (n =12) .166 (n=18).444 (n=18).444 

glossed (n = 12) 1.833 (n = 17) 2.588 (n = 13) 1.615 



A preliminary analysis of these percentages 
suggests that the subjects with the glossed texts 
did not recall significantly more of the glossed 
material than did the unglossed groups. 
Therefore, we might make the assumption that 
for this text, the glossed material was no more 
easily comprehended than the rest of the text. 

Regarding the third research question, con- 
cerning the words in the text that the subjects 
found difficult, the fifteen or fewer words that 
each subject underlined were tallied and ana- 
lyzed. Tables 5, 6, and 7 provide a list of the 
words that were most frequently selected for each 
semester level group and the number of subjects 
that selected them. The uppercase type indicates 
that the word was glossed in the original passage. 



Table 5 



Most Frequently Selected Words By Group - First 
Semester 
First 

semester un^ossed 



EPARGa^ENT 9 

deviennent 6 

jeunes 6 

vraiment 5 

POIDS 5 

moyenne 5 

tot 5 

surtout 4 

copains 4 

CEPENDANT 4 



First 

semester glossed 



vraiment 6 

tot 6 

apparait 4 

chaque 4 

copains 4 

loisir 4 

moyenne 4 

moyens 4 



Table 6 








Most Frequently Selected Words By Group - 


Second Semester 








Second 




Second 




semester unglossed 


semester glossed 




EPARGNENT 


11 


foyer 


11 


DELAISSEES 


10 


EPARGNENT 


7 


PEAU 


8 


apparait 


6 


loisir 


7 


reves 


6 


dehors 


7 


moyenne 


6 


CEPENDANT 


7 


CEPENDANT 


6 


POIDS 


6 


deviennent 


5 


decouverte 


6 


decouverte 


5 


copains 


5 


doute 


4 


Emissions 


5 


QUANTA 


4 


6voluent 


5 


tot 


4 


moyenne 


5 


S ‘ASSUMER 


4 


partage 


5 


copains 


4 


tot 


5 


partage 


4 


vraiment 


4 


acqui^rent 


4 


richesse 


4 






tendent 


4 






plutot 


4 






apparait 


4 






majeure 


4 






done 


4 






int6ret 


4 







Table 7 








Most Frequently Selected Words By Group - 


Third 


Semester 








Third 




Third 




semester unglossed 


semester glossed 




EPARGNENT 


14 


DELAISSEES 


9 


CEPENDANT 


8 


EPARGNENT 


6 


DELAISSEES 


8 


acqui^rent 


6 


PEAU 


8 


paradoxale 


4 


consommation - 


7 


POIDS 


4 


POIDS 


6 


PEAU 


4 


SENTENT 


6 


OUTIL 


4 


I’avenir 


6 


moyenne 


4 


deviennent 


5 


foyer 


4 


decouverte 


5 


4voluent 


4 


partiel 


5 






partage 


4 






autonomie 


4 






int^ret 


4 






SEIN/AU SEIN 


4 






dehors 


4 
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Several interesting differences can be noted 
between the words that the unglossed and the 
glossed groups selected. First, nine of the sub- 
jects in the first semester unglossed group chose 
the word “epargnent”, whereas none of the sub- 
jects in the glossed group did. Since the word had 
been glossed for them, perhaps they were reluc- 
tant to choose it and “use up” one of the fifteen 
words they were entitled to underline. The data 
also shows that the unglossed group chose three 
of the originally glossed words rather frequently 
(“epargnent”, “poids”, and “cependant”), whereas 
none of the originally glossed words appears on 
the glossed group’s list. 

As for the words that were not glossed in the 
original text, it appears that the following four 
words: “vraiment”, “tot”, moyenne”, and “copains” 
were selected rather often by both the glossed and 
unglossed groups. Given that the text was found 
in an intermediate level textbook, it is not sur- 
prising that the first semester students would 
choose words different from those that were 
glossed for the intermediate level students. 



Table 6 show the words that were frequently 
selected by the second semester subjects. In the 
second semester unglossed group, five of the origi- 
nally glossed words were underlined frequently 
and the glossed group chose four of these words 
fairly often. It is notable that unlike the first 
semester glossed group, the second semester 
glossed group did choose words that had been 
originally glossed for them in the passage. 



Regarding the unglossed words, there are sev- 
eral that were frequently indicated by both sec- 
ond semester subject groups as difficult: “decou- 
verte”, “copains”, “partage”, “moyenne”, “appa- 
rait”, and “tot”. Looking back at the first semes- 
ter subjects’ data, three out of four of the words 
that they frequently underlined were also select- 
ed by both groups among the second semester 
learners: “copains”, “tot”, and “moyenne”. 



Table 7 shows the word selection data for the 
third semester groups. Seven of the originally 
glossed words were frequently underlined by the 
unglossed group and five were chosen by the 
glossed group. These two groups both frequently 
selected four of the originally glossed words: 
Epargnent”, “delaissees”, “peau”, and “poids”. 



ment between the lists of the first and second 
semester subjects regarding unglossed words, the 
third semester subjects did not uniformly find 
words that were difficult but had not been glossed 
for them. Since the text was indeed intended for 
the use of third semester French students, these 
results could show that the textbook authors were 
correct in choosing the words for glossing that the 
third semester learners identified as difficult. It 
seems, however, that the words that the subjects 
of all three semester standings chose frequently 
were typically low- frequency words or words that 
were not cognates of English words. 

i 

The above data suggests that the textbook 
authors did indeed gloss many of the words that 
seemed the most difficult to learners of French at 
the second and third semester level. As shown in 
Table 8, the second semester groups indicated 
that 60 to 80 percent (unglossed and glossed 
groups, respectively) of the words that were 
glossed in the original text were difficult for them. 
The third semester groups deemed 86.66 and 
86.66 percent of the originally glossed words as 
problematic. 



Table 8 

Percentage of Subject - Selected Words that Were 
Glossed in the Original Text 

First Second Third 

semester semester semester 

unglossed (n = 12) 46.66 (n = 18) 60.00 (n = 18) 86.66 

glossed (n = 12) 33.33 (n = 17) 80.00 (n = 13) 86.66 



This exploratory analysis supports Brutten’s 
findings that experienced teachers and textbook 
authors are capable of determining which words 
are going to be difficult for readers of foreign lan- 
guage texts for their audience’s proficiency level. 
The fact that the number of glossed words select- 
ed by first and second semester subjects were 
lower may be explained by the fact that since this 
was a text glossed for third semester students, 
there were simply more words that they found 
difficult than the third semester students found 
difficult, thus compelling them to underline differ- 
ent words. 



None of the unglossed words appears on both 
the third semester unglossed and glossed sub- 
jects’ lists of most frequently chosen words. This 
fact suggests that where there was some agree- 

O 

ERIC 



The lists of words that were underlined by the 
subjects of each level, especially those of the third 
semester, for whose level the text was glossed, 
suggests that students may need to have more 
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words glossed for them than are currently being 
glossed. Further research should determine the 
appropriate percentage of words to be glossed in 
texts for second language learners of all levels. 

Limitations and Suggestions 

While recall protocols have been endorsed by 
reading researchers such as Bernhardt as an 
effective method of testing reading comprehen- 
sion, other studies such as that of Wolf have sug- 
gested that there is no one true existing method 
of assessment for reading comprehension can 
have an impact on the results of a study (318). It 
may therefore be beneficial to use more than one 
method of reading comprehension assessment in 
further research. Future studies could perhaps 
include a design similar to the one presented in 
this paper, using several types of reading compre- 
hension assessment methods to make sure that 
the true comprehension of the subjects is being 
measured. Recall protocols have also been criti- 
cized for not being capable of displa 3 dng every- 
thing subjects understand, but simply what they 
can recall. It may be the case that subjects do 
understand much more than they can recall in a 
recall protocol task. 

Text type must also be considered when per- 
forming research on reading- compr^ehsionv-^ 
Davis and Luo found that marginal glosses in the 
subjects’ LI aided comprehension of a literary 
text. However, in this study, there were no signifi- 
cant effects for enhanced comprehension of a jour- 
nalistic text with marginal glosses. Future 
research might compare the effects of glossing for 
several different text types. Perhaps we will then 
discover if glosses are more helpful in under- 
standing certain types of text over others. 

Finally, more studies should also be conducted 
with pedagogical materials to determine if we 
cannot find an even more precise match between 
textbook authors and students as* to which words 
are difficult to understand. Information as to the 
number of words that should be glossed in a text 
for maximum student benefit should also be 
obtained through further research. 

Conclusion 

The results of this study suggest that where 
there was approaching significance for semester 
level for the recall protocol task, there were no 
Significant main effects for glossing condition. 
The glossed groups did not seem to recall signifi- 
cantly higher amounts of glossed material in their 



recalls either. As for the third research question, 
however, the data suggest that there is a rather 
strong match between the words that the subjects 
found difficult in the text and the words that were 
originally glossed in the passage at the second 
and third semester level. Since this text was 
intended for the use. of third semester French stu- 
dents, preliminary analysis suggests that the 
materials developers of this passage were fairly 
accurate with their choice of words to gloss. 

It is important that future research continue 
to examine the value of glossing in foreign lan- 
guage texts. In doing this, we can hopefully 
obtain more knowledge about what aids foreign 
language students in comprehending texts in gen- 
eral. Since so much linguistic, general, and cul- 
tural knowledge can be learned through reading 
authentic texts, we want to make sure that our 
students can use them to their fullest potential. 



*The author would like to thank Professors 
William Glass and Rebecca Kline for their valu- 
able assistance in the execution and writing of 
this study. 



^ '^Rogers and Medley define authentic material 
as “language samples, either oral or written, 
that reflect a naturalness of form and an 
appropriateness of cultural and situational 
context that would be foimd in the language 
as used by native speakers” (468). 

^Bragger, Jeannette D., and Donald B. Rice. 
Allons voir les Frangais et les Frangophones 
(Boston, MA: Heinle and Heinle, 1992) 174. 

^The text was divided into 107 idea imits, 
according to Johnson’s system of breath or 
pausal unit scoring (qtd. in Bernhardt: 209). 
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